Hybrid visual servoing method based on fusion of distance space and image feature space

ABSTRACT

The present disclosure relates to a visual servoing method based on fusion of distance space and image feature space. The method includes: constructing an image Jacobian matrix; constructing a depth Jacobian matrix; fusing the image Jacobian matrix and the depth Jacobian matrix, to obtain a hybrid visual servoing Jacobian matrix; and obtaining a robot motion control amount according to the hybrid visual servoing Jacobian matrix by using a control algorithm, to enable a robot to move to a target position, to complete precise positioning.

RELATED APPLICATION

This application claims priority under 35 U.S.C. 119(a)-(d) to ForeignApplication No. 202010099117.5 entitled “HYBRID VISUAL SERVOING METHODBASED ON FUSION OF DISTANCE SPACE AND IMAGE FEATURE SPACE” and filed inChina on Feb. 18, 2020, the contents of which are herein incorporated intheir entirety by reference for all purposes.

TECHNICAL FIELD

The present disclosure relates to the technical field of visualservoing, and in particular, to a hybrid visual servoing method based onfusion of distance space and image feature space.

BACKGROUND

A visual servoing technology is one of research hotspots in theintelligent robot development field and has been widely used inindustry. To enable a robot to complete more complex work, the robotneeds to have a control system with good performance and a capability ofsensing complex environment changes.

In a conventional control method, control precision of position-basedvisual servoing (PBVS) depends on calibration results of a model and acamera, there is relatively poor robustness, and calibration needs to beusually performed. Compared with the PBVS, image-based visual servoing(IBVS) is more robust in terms of a model parameter and image noise. Ina control method based on development of IBVS, two-dimensional imagefeature information is obtained by using a visual sensor in mostalgorithms, and then the obtained image information is used to completeservoing control. For a servoing task in three-dimensional space,information about an end posture of a robot is lacked. Consequently, anenvironment cannot be comprehensively perceived, and a servoing effectcannot be further improved.

To enable the robot to perceive the environment more comprehensively, tocompensate for information collected by a visual system of the robot, aplurality of sensors can be added to the visual system of the robot, toovercome deficiency in the visual system of the robot. Nowadays, adistance measurement sensor with high precision has been developedrapidly, to provide a new environmental detection scheme. The distancemeasurement sensor and the camera are fused, so that high-precisionpositioning can be provided for a position and posture of a targetobject. However, problems of information fusion and redundancy of thesystem need to be solved due to the plurality of sensors.

For the above background and problems, a new visual servoing methodwould be helpful. A distance measurement sensor and a visual sensor arecombined, to implement mutual supplement of distance space informationand image feature space information, and resolve problems ofmulti-information fusion and information redundancy, so that a robot canperceive an environment change comprehensively, realize precisepositioning of a target object, and complete a servoing task.

SUMMARY

The present application discloses a hybrid visual servoing method andrelated techniques based on fusion of distance space and image featurespace, to resolve problems of multi-information fusion and informationredundancy, so that a robot can perceive an environment changecomprehensively, realize precise positioning of a target object, andcomplete a servoing task.

In some embodiments, a visual servoing method based on fusion ofdistance space and image feature space includes:

constructing an image Jacobian matrix;

constructing a depth Jacobian matrix;

fusing the image Jacobian matrix and the depth Jacobian matrix, toobtain a hybrid visual servoing Jacobian matrix; and

obtaining a robot motion control amount according to the hybrid visualservoing Jacobian matrix by using a control algorithm, to enable a robotto move to a target position, to complete precise positioning.

Optionally, the constructing an image Jacobian matrix specificallyincludes:

obtaining a target image;

converting the target image into a grayscale image, and performingbinarization processing, to obtain a binary image;

performing mean filtering on the binary image, and performing edgedetection on the filtered binary image, to obtain an edge-detectedimage;

detecting the edge-detected image by using a Hough line detectionalgorithm, to obtain two longest straight lines that are intersected;and restoring the two straight lines to the target image, to obtain anintersection point of the two straight lines, where two coordinatepoints of one straight line are (a_(x),a_(y)),(b_(x),b_(y)), twocoordinate points of the other straight line are(c_(x),c_(y)),(d_(x),d_(y)) and the intersection point is represented byO(u,v);

determining a desired posture deflection angle of an end of the robotrelative to a target object according to the intersection point and apoint on one straight line, where the deflection angle is represented byθ; and

determining the image Jacobian matrix according to the intersectionpoint and the deflection angle.

Optionally, the constructing a depth Jacobian matrix specificallyincludes:

determining a position vector of an intersection point of the i^(th)sensor and a measured plane in an apparatus coordinate system;

determining a normal vector of the measured plane according to theposition vector of the intersection point of the i^(th) sensor and themeasured plane in the apparatus coordinate system; and

determining the depth Jacobian matrix according to the normal vector.

Optionally, the obtaining a robot motion control amount according to thehybrid visual servoing Jacobian matrix by using a control algorithm, toenable a robot to move to a target position, to complete precisepositioning specifically includes:

determining an error value of a visual servoing system;

determining an end velocity of the robot according to the error value;and

enabling the robot to move to the target position according to the endvelocity of the robot, to complete precise positioning.

Optionally, the determining a desired posture deflection angle of an endof the robot relative to a target object according to the intersectionpoint and a point on one straight line is specifically expressed as:

$\theta = {a{{\tan\left( \frac{a_{x} - u}{a_{y} - v} \right)}.}}$

Optionally, the determining the image Jacobian matrix according to theintersection point and the deflection angle is specifically expressedas:

${\begin{bmatrix}u \\\overset{˙}{v} \\\overset{˙}{\theta}\end{bmatrix} = {{J_{1}V} = {\begin{bmatrix}{- \frac{\lambda}{z}} & 0 & \frac{u}{z} & \frac{uv}{\lambda} & {- \frac{\lambda^{2} + u^{2}}{z}} & v \\0 & {- \frac{\lambda}{z}} & \frac{v}{z} & \frac{\lambda^{2} + v^{2}}{z} & {- \frac{uv}{\lambda}} & {- u} \\0 & 0 & 0 & 0 & 0 & {- 1}\end{bmatrix}V}}},$

where λ is a focus of a camera, u,v are separately horizontalcoordinates and vertical coordinates of a target point in the image, zis a depth value of the target point in a coordinate system of thecamera, J₁ is the image Jacobian matrix, V=[v_(x), v_(y), v_(z), ω_(x),ω_(y), ω_(z)]^(T), V is an apparatus velocity, and v_(x), v_(y), v_(z),ω_(x), ω_(y), ω_(z) are separately linear velocities and angularvelocities along an x axis, a y axis, and a Z axis.

Optionally, the determining a position vector of an intersection pointof the ith sensor and a measured plane in an apparatus coordinate systemspecifically includes:

${p_{h_{1}}^{Z} = \begin{bmatrix}{\frac{\sqrt{3}}{3}d} \\0 \\h_{1}\end{bmatrix}},{p_{h_{2}}^{Z} = {{R_{z}\left( {\frac{2}{3}\pi} \right)}p_{h_{1}}^{Z}}},{{{and}p_{h_{3}}^{Z}} = {{R_{z}\left( {{- \frac{2}{3}}\pi} \right)}p_{h_{1}}^{Z}}},$where P_(h) ₁ ^(z) is a position vector of an intersection point of thefirst sensor and the measured plane in the apparatus coordinate system,P_(h2) ^(z) is a position vector of an intersection point of the secondsensor and the measured plane in the apparatus coordinate system, P_(h)₃ ^(z) is a position vector of an intersection point of the third sensorand the measured plane in the apparatus coordinate system, d is a fixeddistance between center axes of the three distance measurement sensors,the three sensors are distributed as an equilateral triangle, eachdistance is d, and R_(z) is a rotation matrix around a central axis ofthe camera.

Optionally, the determining a normal vector of the measured planeaccording to the position vector of the intersection point of the ithsensor and the measured plane in the apparatus coordinate system isspecifically expressed as:

$n^{Z} = {\begin{bmatrix}{h_{3} - h_{1} + h_{2} + h_{1}} \\{\sqrt{3}\left( {h_{3} - h_{1} + h_{2} + h_{1}} \right)} \\{\sqrt{3}d}\end{bmatrix} = {\begin{bmatrix}{h_{3} + h_{2} - {2h_{1}}} \\{\sqrt{3}\left( {h_{3} - h_{2}} \right)} \\{\sqrt{3}d}\end{bmatrix}.}}$

Optionally, the determining the depth Jacobian matrix according to thenormal vector is specifically expressed as:

${\begin{bmatrix}\overset{.}{h_{1}} \\\overset{.}{h_{2}} \\\overset{.}{h_{3}}\end{bmatrix} = {{J_{2}V} = {\begin{bmatrix}{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} & {{- \frac{\sqrt{3}}{3d}}\left( p_{h_{1}}^{z} \right)^{T}{S\left( n^{Z} \right)}} \\{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} & {{- \frac{\sqrt{3}}{3d}}\left( p_{h_{2}}^{z} \right)^{T}{S\left( n^{Z} \right)}} \\{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} & {{- \frac{\sqrt{3}}{3d}}\left( p_{h_{3}}^{z} \right)^{T}{S\left( n^{Z} \right)}}\end{bmatrix}V}}},$where S(n^(z)) is antisymmetric matrix of n^(z).

Optionally, the determining an end velocity of the robot according tothe error value is specifically expressed as:V=−KJ ⁺ e,where J⁺ is an inverse matrix of a hybrid visual Jacobian matrix, e isan error value,

${e = {f_{d} - f}},\left\{ {\begin{matrix}{\overset{.}{e} = {- {Ke}}} \\{\overset{.}{e} = {JV}}\end{matrix},} \right.$K is a proportionality coefficient, and V is an end apparatus velocityof the robot.

According to specific embodiments of the present disclosure, the presentdisclosure has the following technical effects:

Distance information obtained by a distance measurement sensor in thepresent disclosure compensates for insufficient feedback information ofa visual sensor. The distance information and visual information iscombined, to achieve full measurement of three-dimensional space, andavoid redundancy during information fusion. In the present disclosure, ahybrid Jacobian matrix formed through space fusion is a 6-dimensionalfull-rank matrix. A pseudo-inverse solution of a matrix is avoided,problems of a local minimum and singularity in conventional IBVS areresolved, and a system is globally asymptotically stable. In a visualservoing process, the system can track the target point accurately andquickly. In addition, the method provides a new research direction fordevelopment of visual servoing control.

BRIEF DESCRIPTION OF DRAWINGS

To describe the techniques described in the embodiments art moreclearly, the accompanying drawings illustrating the embodiments will bedescribed briefly below. A person of ordinary skill in the art may stillderive other drawings from these accompanying drawings without creativeefforts.

FIG. 1 is a flowchart of a visual servoing method based on fusion ofdistance space and image feature space according to an embodimentdisclosed herein;

FIG. 2 is a plan view of a robot, an apparatus, and a target objectaccording to an embodiment disclosed herein;

FIG. 3 is a schematic diagram of image feature space informationaccording to an embodiment disclosed herein;

FIG. 4A is a schematic diagram of sensor installation according to anembodiment disclosed herein;

FIG. 4B is another schematic diagram of sensor installation according toan embodiment disclosed herein;

FIG. 5 is a schematic diagram of a field of view of a camera accordingto an embodiment disclosed herein; and

FIG. 6 is a diagram of a visual servoing result according to anembodiment disclosed herein.

FIG. 7 is a block diagram that illustrates a computer system upon whichan embodiment of the approach may be implemented.

DETAILED DESCRIPTION

The following describes various embodiments with reference toaccompanying drawings. Additional embodiments obtained by a person ofordinary skill in the art based on the examples in the presentapplication without creative efforts shall fall within the scope of thepresent application.

The present application discloses a hybrid visual servoing method basedon fusion of distance space and image feature space, to resolve problemsof multi-information fusion and information redundancy, so that a robotcan perceive an environment change comprehensively, realize precisepositioning of a target object, and complete a servoing task.

The various embodiments are further described in detail below withreference to the accompanying drawings and specific implementations.

FIG. 2 is a plan view of a robot, an apparatus, and a target objectaccording to an embodiment of the present disclosure. Specifically, FIG.2 shows the installation relationship between the apparatus and therobot and the spatial pose of the target object relative to the robot.

In some embodiments, a target center (u, v, θ) is first obtained byusing a visual sensor and a related image processing algorithm. Herein,(u,v) represents image coordinates of a center of the target object, andθ represents a deflection angle that is of image information between anend of the robot and an object and that is relative to a desiredposture, as illustrated in FIG. 3 . Distance information (h₁, h₂, h₃) isobtained by using three distance measurement sensors, as illustrated inFIG. 4A and FIG. 4B, and a combination of the visual information and thedistance information is used as a feature in visual servoing, to obtaina feature Jacobian matrix (which is used to describe a velocityrelationship between a feature and an apparatus fixed at an end of therobot).

In some embodiments, a corresponding control algorithm is designed, toobtain a robot motion control amount, to enable a robot to move to atarget position, to complete a precise positioning task.

FIG. 1 is a flowchart illustrating a visual servoing method based onfusion of distance space and image feature space according to anembodiment of the present disclosure. As shown in FIG. 1 , the methodincludes:

Step 101: Construct an image Jacobian matrix.

In some embodiments, V=[v_(x), v_(y), v_(z), ω_(x), ω_(y), ω_(z)]^(T) isused to represent an apparatus velocity, and respectively includeslinear velocities and angular velocities along an x axis, a y axis, anda Z axis, f=[u, v, θ, h₁, h₂, h₃]^(T) is used to represent a systemfeature, and respectively includes horizontal coordinates and verticalcoordinates of a target object in an image, a deflection angle that isbetween an end of the robot and the object and that is relative to adesired posture, and distance measurement values of three distancemeasurement sensors, and f_(d)=[u_(d), v_(d), θ_(d), h_(1d), h_(2d),h_(3d)]^(T) is used to represent a desired system feature.

First, the image is obtained by using a common RGB industrial cameraintegrating an end apparatus.

The target image is converted into a grayscale image, and binarizationprocessing is performed, to obtain a binary image.

Mean filtering is performed on the binary image, to remove a noise pointfrom the image, and edge detection is performed on the filtered binaryimage by using a Canny algorithm or other edge detection algorithmsknown to someone skilled in the art, to obtain an edge-detected image.

The edge-detected image is detected by using a Hough line detectionalgorithm or other line detection methods known to someone skilled inthe art. During detection, different quantities of straight lines areobtained according to different straight-line lengths and confidencelevel parameters. The parameters (a straight-line length and aconfidence level) are adjusted according to an actual length of thetarget object. In addition, two longest straight lines that areintersected can be obtained according to a condition that two straightlines are intersected. The straight lines are drawn on an originalimage, as shown in FIG. 5 and FIG. 6 .

The two straight lines obtained in the previous step are given by usingcoordinates of start points and coordinates of end points of thestraight lines. It is assumed that the two straight lines that areintersected are A and B, two coordinate points of the straight line Aare (a_(x),a_(y)),(b_(x),b_(y)) and coordinate points of the straightline B are (c_(x), c_(d)),(d_(x),d_(y)). Therefore, an intersectionpoint of the two straight lines may be represented as O(u,v):u=(a _(x) +a ₀ ·r ₁₁ +c _(x) +a ₁ ·r ₂₁)/2v=(a _(y) +a ₀ ·r ₁₂ +c _(y) +a ₁ ·r ₂₂)/2

Herein r₁₁=b_(x)−a_(x), r₁₂=b_(y)−a_(y), r₂₁=c_(x)−d_(x),r₂₂=c_(y)−d_(y), and

$\begin{bmatrix}a_{0} \\a_{1}\end{bmatrix} = {{\begin{bmatrix}{b_{x} - a_{x}} & {- \left( {c_{x} - d_{x}} \right)} \\{b_{y} - a_{y}} & {- \left( {c_{y} - d_{y}} \right)}\end{bmatrix}^{- 1}\begin{bmatrix}{c_{x} - a_{x}} \\{c_{y} - a_{y}}\end{bmatrix}}.}$

The deflection angle of the end of the robot relative to the desiredposture of target object is determined according to the intersectionpoint and a point on one straight line. The deflection angle isrepresented by θ:

$\theta = {a{{\tan\left( \frac{a_{x} - u}{a_{y} - v} \right)}.}}$

The image Jacobian matrix is determined according to the intersectionpoint and the deflection angle.

The intersection point (u,v) of image features is image coordinates(u,v) of a target center, and an included angle θ is the deflectionangle that is between the end and the object and that is relative to thedesired posture. Further, an image Jacobian matrix J₁ used to describe acamera velocity and a feature change velocity ({dot over (u)}, {dot over(v)}, {dot over (θ)}) is obtained according to a known quantity:

$\begin{bmatrix}\overset{.}{u} \\\overset{.}{v} \\\overset{.}{\theta}\end{bmatrix} = {{J_{1}V} = {\begin{bmatrix}{- \frac{\lambda}{z}} & 0 & \frac{u}{z} & \frac{uv}{\lambda} & {- \frac{\lambda^{2} + v^{2}}{\lambda}} & v \\0 & {- \frac{\lambda}{z}} & \frac{v}{z} & \frac{\lambda^{2} + v^{2}}{\lambda} & {- \frac{uv}{\lambda}} & {- u} \\0 & 0 & 0 & 0 & 0 & {- 1}\end{bmatrix}{V.}}}$

Herein, λ is a focus of a camera, u, v are separately horizontalcoordinates and vertical coordinates of a target point in the image, zis a depth value of the target point in a coordinate system of thecamera, V=[v_(x), v_(y), v_(z), ω_(x), ω_(y), ω_(z)]^(T), V is anapparatus velocity, and v_(x), v_(y), v_(z), ω_(x), ω_(y), ω_(z) areseparately linear velocities and angular velocities along an x axis, a yaxis, and a Z axis.

Step 102: Construct a depth Jacobian matrix.

In some embodiments, the distance measurement sensors are assembled asshown in FIG. 4A. The three distance measurement sensors are installedin an equilateral triangle by using an axial direction of a visualsensor as a central axis, and distances between axial directions of thethree distance measurement sensors have fixed values and are equal. Thethree distance measurement sensors are configured to: collect distanceinformation in real time, and calculate the depth Jacobian matrix.

A position vector of an intersection point of the ith sensor and ameasured plane in an apparatus coordinate system is determined:

${p_{h_{1}}^{Z} = \begin{bmatrix}{\frac{\sqrt{3}}{3}d} \\0 \\h_{1}\end{bmatrix}},{p_{h_{2}}^{Z} = {{R_{z}\left( {\frac{2}{3}\pi} \right)}p_{h_{1}}^{Z}}},{{{and}p_{h_{3}}^{Z}} = {{R_{z}\left( {{- \frac{2}{3}}\pi} \right)}{p_{h_{1}}^{Z}.}}}$Herein, P_(h) ₁ ^(z) is a position vector of an intersection point ofthe first sensor and the measured plane in the apparatus coordinatesystem, P_(h2) ^(z) is a position vector of an intersection point of thesecond sensor and the measured plane in the apparatus coordinate system,P_(h) ₃ ^(z) is a position vector of an intersection point of the thirdsensor and the measured plane in the apparatus coordinate system, d is afixed distance between center axes of the three distance measurementsensors, the three sensors are distributed as an equilateral triangle,each distance is d, and R_(z) is a rotation matrix around a central axisof the camera.

A normal vector of the measured plane is determined according to theposition vector of the intersection point of the ith sensor and themeasured plane in the apparatus coordinate system:

$n^{Z} = {\begin{bmatrix}{h_{3} - h_{1} + h_{2} + h_{1}} \\{\sqrt{3}\left( {h_{3} - h_{1} + h_{2} + h_{1}} \right)} \\{\sqrt{3}d}\end{bmatrix} = \begin{bmatrix}{h_{3} + h_{2} - {2h_{1}}} \\{\sqrt{3}\left( {h_{3} - h_{2}} \right)} \\{\sqrt{3}d}\end{bmatrix}}$

Therefore, a Jacobian matrix J₂ between a depth measurement value h_(i)(i=1, 2, 3) and an apparatus velocity is obtained.

$\begin{bmatrix}\overset{.}{h_{1}} \\\overset{.}{h_{2}} \\\overset{.}{h_{3}}\end{bmatrix} = {{J_{2}V} = {\begin{bmatrix}{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} & {{- \frac{\sqrt{3}}{3d}}\left( p_{h_{1}}^{Z} \right)^{T}{S\left( n^{Z} \right)}} \\{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} & {{- \frac{\sqrt{3}}{3d}}\left( p_{h_{2}}^{Z} \right)^{T}{S\left( n^{Z} \right)}} \\{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} & {{- \frac{\sqrt{3}}{3d}}\left( p_{h_{3}}^{Z} \right)^{T}{S\left( n^{Z} \right)}}\end{bmatrix}{V.}}}$Herein, S(n^(z)) is an antisymmetric array of n^(z).

It is assumed that n^(z)=[n1, n2, n3]^(T). Herein, the antisymmetricarray of n^(z) is

${S\left( n^{Z} \right)} = {\begin{bmatrix}0 & {{- n}3} & {n2} \\{n3} & 0 & {{- n}1} \\{{- n}2} & {n1} & 0\end{bmatrix}.}$

Step 103: Fuse the image Jacobian matrix and the depth Jacobian matrix,to obtain a hybrid visual servoing Jacobian matrix.

$\overset{.}{f} = {\begin{bmatrix}\overset{.}{u} \\\overset{.}{v} \\\overset{.}{\theta} \\\overset{.}{h_{1}} \\\overset{.}{h_{2}} \\\overset{.}{h_{3}}\end{bmatrix} = {{\begin{bmatrix}J_{1} \\J_{2}\end{bmatrix}V} = {\begin{bmatrix}{\begin{matrix}\begin{matrix}{- \frac{\lambda}{z}} & 0 & \frac{u}{z} & \frac{uv}{\lambda} & {- \frac{\lambda^{2} + v^{2}}{\lambda}}\end{matrix} \\\begin{matrix}0 & {- \frac{\lambda}{z}} & \frac{v}{z} & \frac{\lambda^{2} + v^{2}}{\lambda} & {- \frac{uv}{\lambda}} & {- u}\end{matrix} \\\begin{matrix}0 & 0 & 0 & 0 & 0 & {- 1}\end{matrix}\end{matrix}} \\\begin{matrix}\begin{matrix}{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} \\{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} \\{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}}\end{matrix} & \begin{matrix}{{- \frac{\sqrt{3}}{3d}}\left( p_{h_{1}}^{Z} \right)^{T}{S\left( n^{Z} \right)}} \\{{- \frac{\sqrt{3}}{3d}}\left( p_{h_{2}}^{Z} \right)^{T}{S\left( n^{Z} \right)}} \\{{- \frac{\sqrt{3}}{3d}}\left( p_{h_{3}}^{Z} \right)^{T}{S\left( n^{Z} \right)}}\end{matrix}\end{matrix}\end{bmatrix}V}}}$

It can be learned, through analysis, that the Jacobian matrix is a6-dimensional full-rank matrix, and meets JJ⁺>0. Therefore, a system isglobally asymptotically stable.

Step 104: Obtain a robot motion control amount according to the hybridvisual servoing Jacobian matrix by using a control algorithm, to enablea robot to move to a target position, to complete precise positioning.

In some embodiments, the corresponding control algorithm is designed,and the Jacobian matrix is combined, to provide a velocity controlamount of the end apparatus. Herein,f_(d)=[u_(d),v_(d),θ_(d),h_(1d),h_(2d),h_(3d)]^(T) represents a desiredsystem feature, and is a measurement value of each feature when thesystem reaches the target position.

An error value e=f_(d)−f of a visual servoing system is determined. Itis assumed that

$\left\{ {\begin{matrix}{\overset{˙}{e} = {- {Ke}}} \\{\overset{˙}{e} = {JV}}\end{matrix},} \right.$K is a proportional coefficient, and V is a velocity of an end apparatusof the robot, that is, an end velocity of the robot.

The end velocity of the robot is determined according to the errorvalue:V=−KJ ⁺ e,where J⁺ is an inverse matrix of a hybrid visual Jacobian matrix, and eis an error value.

The robot is enabled to move to the target position according to the endvelocity of the robot, to complete precise positioning:

After the end velocity V of the robot is obtained, the end velocity isdirectly sent to the robot, so that the robot moves. In addition, whenthe robot reaches the target position, 0 is obtained according toe=f_(d)−f. Therefore, the calculated V is 0. In this case, the robotstops moving. That is, after the robot moves to the target position, therobot stops moving, to complete a precise positioning task.

According to the 6-dimensional full-rank Jacobian matrix in the hybridvisual servoing method in the present disclosure, problems such assingularity and a local minimum of a conventional Jacobian matrix areavoided. Complementary image feature space and distance space are fused.A deficiency in visual information is compensated for without redundantinformation. A system collects data in real time, and can quicklycomplete a visual servoing task according to a simple control algorithm.

Each embodiment of this specification is described in an incrementalmanner. Each embodiment focuses on the difference from otherembodiments. Certain portions of one embodiment could be combined withor could replace other portions of another embodiment when applicable.

In this specification, specific examples are used to describe theprinciples and implementations of the embodiments. The description ofthe foregoing embodiments is used to help understand the method of thepresent disclosure and related techniques. In addition, a person ofordinary skill in the art can make various modifications in terms ofspecific implementations and scope of application in accordance with theideas of the present disclosure. In conclusion, the content of thisspecification shall not be construed as a limitation to the presentdisclosure.

Hardware Overview

According to one embodiment, the techniques described herein areimplemented by one or more special-purpose computing devices. Thespecial-purpose computing devices may be hard-wired to perform thetechniques, or may include digital electronic devices such as one ormore application-specific integrated circuits (ASICs) or fieldprogrammable gate arrays (FPGAs) that are persistently programmed toperform the techniques, or may include one or more general purposehardware processors programmed to perform the techniques pursuant toprogram instructions in firmware, memory, other storage, or acombination. Such special-purpose computing devices may also combinecustom hard-wired logic, ASICs, or FPGAs with custom programming toaccomplish the techniques. The special-purpose computing devices may bedesktop computer systems, portable computer systems, handheld devices,networking devices or any other device that incorporates hard-wiredand/or program logic to implement the techniques.

For example, FIG. 7 is a block diagram that illustrates a computersystem 700 upon which an embodiment of the approach may be implemented.Computer system 700 includes a bus 702 or other communication mechanismfor communicating information, and a hardware processor 704 coupled withbus 702 for processing information. Hardware processor 704 may be, forexample, a general purpose microprocessor.

Computer system 700 also includes a main memory 706, such as a randomaccess memory (RAM) or other dynamic storage device, coupled to bus 702for storing information and instructions to be executed by processor704. Main memory 706 also may be used for storing temporary variables orother intermediate information during execution of instructions to beexecuted by processor 704. Such instructions, when stored innon-transitory storage media accessible to processor 704, rendercomputer system 700 into a special-purpose machine that is customized toperform the operations specified in the instructions.

Computer system 700 further includes a read only memory (ROM) 708 orother static storage device coupled to bus 702 for storing staticinformation and instructions for processor 704. A storage device 710,such as a magnetic disk or optical disk, is provided and coupled to bus702 for storing information and instructions.

Computer system 700 may be coupled via bus 702 to a display 712, such asa cathode ray tube (CRT), for displaying information to a computer user.An input device 714, including alphanumeric and other keys, is coupledto bus 702 for communicating information and command selections toprocessor 704. Another type of user input device is cursor control 716,such as a mouse, a trackball, or cursor direction keys for communicatingdirection information and command selections to processor 704 and forcontrolling cursor movement on display 712. This input device typicallyhas two degrees of freedom in two axes, a first axis (e.g., x) and asecond axis (e.g., y), that allows the device to specify positions in aplane.

Computer system 700 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 700 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 700 in response to processor 704 executing one or more sequencesof one or more instructions contained in main memory 706. Suchinstructions may be read into main memory 706 from another storagemedium, such as storage device 710. Execution of the sequences ofinstructions contained in main memory 706 causes processor 704 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may comprisenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 710.Volatile media includes dynamic memory, such as main memory 706. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, and EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire and fiber optics, including thewires that comprise bus 702. Transmission media can also take the formof acoustic or light waves, such as those generated during radio-waveand infra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 704 for execution. For example,the instructions may initially be carried on a magnetic disk or solidstate drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over atelephone line using a modem. A modem local to computer system 700 canreceive the data on the telephone line and use an infra-red transmitterto convert the data to an infra-red signal. An infra-red detector canreceive the data carried in the infra-red signal and appropriatecircuitry can place the data on bus 702. Bus 702 carries the data tomain memory 706, from which processor 704 retrieves and executes theinstructions. The instructions received by main memory 706 mayoptionally be stored on storage device 710 either before or afterexecution by processor 704.

Computer system 700 also includes a communication interface 718 coupledto bus 702. Communication interface 718 provides a two-way datacommunication coupling to a network link 720 that is connected to alocal network 722. For example, communication interface 718 may be anintegrated services digital network (ISDN) card, cable modem, satellitemodem, or a modem to provide a data communication connection to acorresponding type of telephone line. As another example, communicationinterface 718 may be a local area network (LAN) card to provide a datacommunication connection to a compatible LAN. Wireless links may also beimplemented. In any such implementation, communication interface 718sends and receives electrical, electromagnetic or optical signals thatcarry digital data streams representing various types of information.

Network link 720 typically provides data communication through one ormore networks to other data devices. For example, network link 720 mayprovide a connection through local network 722 to a host computer 724 orto data equipment operated by an Internet Service Provider (ISP) 726.ISP 726 in turn provides data communication services through the worldwide packet data communication network now commonly referred to as the“Internet” 728. Local network 722 and Internet 728 both use electrical,electromagnetic or optical signals that carry digital data streams. Thesignals through the various networks and the signals on network link 720and through communication interface 718, which carry the digital data toand from computer system 700, are example forms of transmission media.

Computer system 700 can send messages and receive data, includingprogram code, through the network(s), network link 720 and communicationinterface 718. In the Internet example, a server 730 might transmit arequested code for an application program through Internet 728, ISP 726,local network 722 and communication interface 718.

The received code may be executed by processor 704 as it is received,and/or stored in storage device 710, or other non-volatile storage forlater execution.

In the foregoing specification, embodiments of the approach have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the approach,and what is intended by the applicants to be the scope of the approach,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

What is claimed is:
 1. A visual servoing method based on fusion ofdistance space and image feature space, comprising: constructing animage Jacobian matrix; constructing a depth Jacobian matrix; fusing theimage Jacobian matrix and the depth Jacobian matrix, to obtain a hybridvisual servoing Jacobian matrix; obtaining a robot motion control amountaccording to the hybrid visual servoing Jacobian matrix by using acontrol algorithm; and controlling, in accordance with the obtainedrobot motion control amount, a robot to move to a target position, tocomplete precise positioning; wherein constructing the image Jacobianmatrix comprises: obtaining a target image; converting the target imageinto a grayscale image, and performing binarization processing, toobtain a binary image; performing mean filtering on the binary image,and performing edge detection on the filtered binary image, to obtain anedge-detected image; detecting the edge-detected image by using a Houghline detection algorithm, to obtain two longest straight lines that areintersected; and restoring the two straight lines to the target image,to obtain an intersection point of the two straight lines, wherein twocoordinate points of one straight line are (a_(x),a_(y)), (b_(x),b_(y)),two coordinate points of the other straight line are (c_(x),c_(y)),(d_(x),d_(y)), and the intersection point is represented by O(u,v);determining a deflection angle of an end of the robot relative to adesired posture of a target object according to the intersection pointand a point on one straight line, wherein the deflection angle isrepresented by θ; and determining the image Jacobian matrix according tothe intersection point and the deflection angle.
 2. The visual servoingmethod based on fusion of distance space and image feature spaceaccording to claim 1, wherein the constructing a depth Jacobian matrixspecifically comprises: determining a position vector of an intersectionpoint of an ith sensor and a measured plane in an apparatus coordinatesystem; determining a normal vector of the measured plane according tothe position vector of the intersection point of the ith sensor and themeasured plane in the apparatus coordinate system; and determining thedepth Jacobian matrix according to the normal vector.
 3. The visualservoing method based on fusion of distance space and image featurespace according to claim 2, wherein the determining a position vector ofan intersection point of the ith sensor and a measured plane in anapparatus coordinate system specifically comprises:${p_{h_{1}}^{Z} = \begin{bmatrix}{\frac{\sqrt{3}}{3}d} \\0 \\h_{1}\end{bmatrix}},{p_{h_{1}}^{Z} = {{R_{z}\left( {\frac{2}{3}\pi} \right)}p_{h_{1}}^{Z}}},{{{and}p_{h_{3}}^{Z}} = {{R_{z}\left( {{- \frac{2}{3}}\pi} \right)}p_{h_{1}}^{Z}}},$ wherein P_(h) ₁ ^(z) is a position vector of an intersection point of afirst sensor and the measured plane in the apparatus coordinate system,P_(h2) ^(z) is a position vector of an intersection point of a secondsensor and the measured plane in the apparatus coordinate system, P_(h)₃ ^(z) is a position vector of an intersection point of a third sensorand the measured plane in the apparatus coordinate system, d is a fixeddistance between center axes of the three distance measurement sensors,the three sensors are distributed as an equilateral triangle, eachdistance is d, and R_(z) is a rotation matrix around a central axis ofthe camera.
 4. The visual servoing method based on fusion of distancespace and image feature space according to claim 2, wherein thedetermining a normal vector of the measured plane according to theposition vector of the intersection point of the ith sensor and themeasured plane in the apparatus coordinate system is specificallyexpressed as: ${n^{Z} = {\begin{bmatrix}{h_{3} - h_{1} + h_{2} - h_{1}} \\{\sqrt{3}\left( {h_{3} - h_{1} + h_{2} + h_{1}} \right)} \\{\sqrt{3}d}\end{bmatrix} = \begin{bmatrix}{h_{3} + h_{2} - {2h_{1}}} \\{\sqrt{3}\left( {h_{3} - h_{2}} \right)} \\{\sqrt{3}d}\end{bmatrix}}},$  wherein n^(z) is the normal vector of the measuredplane, h₁ is a depth measurement value of a first sensor, h₂ is a depthmeasurement value of a second sensor, h₃ is a depth measurement value ofa third sensor, and d is a fixed distance between center axes of threedistance measurement sensors.
 5. The visual servoing method based onfusion of distance space and image feature space according to claim 2,wherein the determining the depth Jacobian matrix according to thenormal vector is specifically expressed as: ${\begin{bmatrix}\overset{.}{h_{1}} \\\overset{.}{h_{2}} \\\overset{.}{h_{3}}\end{bmatrix} = {{J_{2}V} = {\begin{bmatrix}{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} & {{- \frac{\sqrt{3}}{3d}}\left( p_{h_{1}}^{Z} \right)^{T}{S\left( n^{Z} \right)}} \\{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} & {{- \frac{\sqrt{3}}{3d}}\left( p_{h_{2}}^{Z} \right)^{T}{S\left( n^{Z} \right)}} \\{{- \frac{\sqrt{3}}{3d}}\left( n^{Z} \right)^{T}} & {{- \frac{\sqrt{3}}{3d}}\left( p_{h_{3}}^{Z} \right)^{T}{S\left( n^{Z} \right)}}\end{bmatrix}V}}},$  wherein h₁ is a depth measurement value of a firstsensor, h₂ is a depth measurement value of a second sensor, h₃ is adepth measurement value of a third sensor, J₂ is a depth Jacobianmatrix, V is an apparatus velocity, n^(z) is the normal vector of themeasured plane, and S(n^(z)) is antisymmetric matrix of n^(z).
 6. Thevisual servoing method based on fusion of distance space and imagefeature space according to claim 1, wherein the obtaining a robot motioncontrol amount according to the hybrid visual servoing Jacobian matrixby using a control algorithm, to enable a robot to move to a targetposition, to complete precise positioning specifically comprises:determining an error value of a visual servoing system; determining anend velocity of the robot according to the error value; and enabling therobot to move to the target position according to the end velocity ofthe robot, to complete precise positioning.
 7. The visual servoingmethod based on fusion of distance space and image feature spaceaccording to claim 6, wherein the determining an end velocity of therobot according to the error value is specifically expressed as:V=−KJ ⁺ e,  wherein J⁺ is an inverse matrix of a hybrid visual Jacobianmatrix, e is an error value, e=f_(d)−f, $\left\{ {\begin{matrix}{\overset{˙}{e} = {- {Ke}}} \\{\overset{˙}{e} = {JV}}\end{matrix},} \right.$  K is a proportionality coefficient, and V is anend apparatus velocity of the robot.
 8. The visual servoing method basedon fusion of distance space and image feature space according to claim1, wherein the determining a deflection angle of an end of the robotrelative to the desired posture of target object according to theintersection point and a point on one straight line is specificallyexpressed as:$\theta = {a{{\tan\left( \frac{a_{x} - u}{a_{y} - v} \right)}.}}$
 9. Thevisual servoing method based on fusion of distance space and imagefeature space according to claim 1, wherein the determining the imageJacobian matrix according to the intersection point and the deflectionangle is specifically expressed as: ${\begin{bmatrix}\overset{.}{u} \\\overset{.}{v} \\\overset{.}{\theta}\end{bmatrix} = {{J_{1}V} = {\begin{bmatrix}{- \frac{\lambda}{z}} & 0 & \frac{u}{z} & \frac{uv}{\lambda} & {- \frac{\lambda^{2} + u^{2}}{\lambda}} & v \\0 & {- \frac{\lambda}{z}} & \frac{v}{z} & \frac{\lambda^{2} + v^{2}}{\lambda} & {- \frac{uv}{\lambda}} & {- u} \\0 & 0 & 0 & 0 & 0 & {- 1}\end{bmatrix}V}}},$  wherein λ is a focus of a camera, u,v areseparately horizontal coordinates and vertical coordinates of a targetpoint in the image, z is a depth value of the target point in acoordinate system of the camera, J₁ is the image Jacobian matrix,V=[v_(x), v_(y), v_(z), ω_(x), ω_(y), ω_(z)]^(T), V is an apparatusvelocity, and v_(x), v_(y), v_(z), ω_(x), ω_(y), ω_(z) are separatelylinear velocities and angular velocities along an x axis, a y axis, anda Z axis.